fix(ollama): make native num_ctx opt-in#2589
Conversation
|
really nice bug report and fix — the repro and the i pushed one small commit ( one thing to confirm before i merge: you're threading it from |
|
Yes, that scope is intentional. For the native Ollama path this PR changes, the override is a process-level config value: The behavior I intended and covered here is:
I chose that scope because the bug was caused by Hindsight forcing a request-level Ollama runner option even when the Ollama deployment already had its own context configuration. If a deployment needs different context sizes per Ollama host/model, I think the safer path is server-side Modelfile settings or separate Hindsight processes rather than one shared process env var. I also pushed Thanks for adding the env-template entries and keeping the generated-files check green. |
Summary
num_ctxunless it is explicitly configuredHINDSIGHT_API_LLM_OLLAMA_NUM_CTXand thread it through config,LLMProvider, andMemoryEnginenum_ctx=16384request behaviorFixes #2572.
Tests
python3 -m py_compile hindsight-api-slim/hindsight_api/config.py hindsight-api-slim/hindsight_api/engine/llm_wrapper.py hindsight-api-slim/hindsight_api/engine/memory_engine.py hindsight-api-slim/hindsight_api/engine/providers/openai_compatible_llm.py hindsight-api-slim/tests/test_config_validation.py hindsight-api-slim/tests/test_llm_wrapper.pyruff check hindsight-api-slim/hindsight_api/config.py hindsight-api-slim/hindsight_api/engine/llm_wrapper.py hindsight-api-slim/hindsight_api/engine/memory_engine.py hindsight-api-slim/hindsight_api/engine/providers/openai_compatible_llm.py hindsight-api-slim/tests/test_config_validation.py hindsight-api-slim/tests/test_llm_wrapper.pyruff format --check hindsight-api-slim/hindsight_api/config.py hindsight-api-slim/hindsight_api/engine/llm_wrapper.py hindsight-api-slim/hindsight_api/engine/memory_engine.py hindsight-api-slim/hindsight_api/engine/providers/openai_compatible_llm.py hindsight-api-slim/tests/test_config_validation.py hindsight-api-slim/tests/test_llm_wrapper.pyTargeted pytest did not run in this local environment:
uv runcannot installonnxruntime==1.27.0for macOS arm64, and the system Python environment is missing project deps such asasyncpg.